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Abstract 

By using variational methods, we study the existence of mountain pass solution 
to the following doubly critical Schrodinger system: 

j— Au — — \u\ 2 *~ 2 u = h(x)a\u\ a ~ 2 \v\^u inR N , 

[—Av — H 2 j^z~\v\ 2 - 2 v = h(x)/3\u\ a \v\^~ 2 v inM N , 

where a > 2, /? > 2, a + /3 < 2*; /xi, ji 2 G [0, )■ The weight function h(x) 

is allowed to be sign-changing so that the nonlinearities include a large class of 
indefinite weights. We show that the PS condition is satisfied at higher energy 
level when a + /3 = 2* and obtain the existence of mountain pass solution. 
Besides, a nonexistence result of the ground state is given. 
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1 Introduction 


In this paper, we investigate the existence of solutions to the following nonlinear 
Schrodinger system: 

f—A m — /Ltippr — \u\ 2 *~ 2 u = h(x)a\u\ a ~ 2 \v\Pu, inR^, 

Am — ~ |u| 2 * -2 u = |vC —2 v, inR^, 

where a > 2, /3 > 2, «+/? < 2*; /zi, /z 2 G [0, Aw), Aw := ; /i(x) G L 00 (R jY ). 

The interest for such systems is motivated by its applications to plasma physics, 
nonlinear optics, condensed matter physics, etc. For example, the coupled non¬ 
linear Schrodinger systems arise in the description of several physical phenom¬ 
ena such as the propagation of pulses in birefringent optical fibers and Kerr-like 
photorefractive media, see [3 M M M [331 HI, etc. Also, it is related to the 
following Gross-Pitaevskii equations (cf. 17, 32]): 


{ -i^<I>i = A$i - a(x)4>i + /zi|$i| 2 $i + z/|$ 2 | 2 $i, x G R^, t > 0, 
-i -§- t ^2 = A4> 2 - b(x)<$> 2 + + ^|$i| 2 4> 2 , x G R^, t > 0, 

$j=J,-(i,t)GC, j = 1,2, 

$j(x,t) —> 0, as |a;| — > +oo, t > 0, j = 1,2, 


where i is the imaginary unit; a(x),b(x) are potential functions. Problem (11.21) 
also arises in the Hartree-Fock theory for a double condensate, i.e., a binary 
mixture of Bose-Einstein condensates in two different hyperfine states (see (13]). 
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We call a solution (u, v) nontrivial if both u ^ 0 and v ^ 0; we call a solution 
(u,v) semi-trivial if (u,v) is a type of (u,0) or (0,u). The existence of serni- 
trivial solution is equivalent to the solution of the following scalar equation: 

— Au — A* 7 - 75 - = M 2 in M^, (1.3) 

FI 

whose solutions have been figured out. Here, when fj, = 0, we refer the readers 
to HUB® [7J[8J [SUSHIS]. When [i G (0, ), by [51] . an explicit solution of 

m exists, namely 


-iF) 


A(N,») 

4 aa N - 2 ' 

\ x \ a ,, (1 _|_ 2 


(1.4) 


N(N-2-2a u ) 2 

N-2 


This solution 


where a M := — y (— 2 ) 2 — n and A(N, fj.) := 

is also known to be the unique positive solution up to a conformal transformation 
of the form 


z£=<r a>0. (1.5) 

G 

Before returning to the existence and nonexistence of the nontrivial solutions of 
m, we recall the very recent paper [I], where the authors studied the existence 
of solutions to the following system: 


f —Au — Mijyp- — \u\ 2 * 2 u = v • h{x)a\u\ a 2 \v\^u, in 1^, 

Au — fj, 2 jyp — |u| 2 *~ 2 u = v ■ h(x)(3\u\ a \v\ l3 - 2 v, in 1^, 

where p,i, /i 2 G (0, ^ 4 ^ ) and the parameter v serves as a regulator. Through¬ 
out the paper [T], h{x) satisfies the following condition: 


Let 


h(x) > 0, h(x) £ 0, h[x) G L 1 ^) n L 00 ^). 

N-l 

4 fH 


S(JH) “ 1 - 


5, 


(N~2) 2 , 

where S is the sharp constant of D 1,2 ( R w ) L 2 (R w ) satisfying 


and 


|Vu | 2 dx > S ( / |u | 2 dx 

JR" 


„ N(N — 2 ) N 2 N(N-2) n * W + 1,-. 

S=— - L\g N \N=—± - L2ntt 1+ ;vr(-) J 

4 4 v 2 ’ 


In particular, S = 3(f) 3 when N = 3; 5 = 4 Y^ 7r when N = 4. 


(1.7) 

( 1 . 8 ) 

(1.9) 

( 1 . 10 ) 
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If the coupling terms are of subcritical case, i.e., a+/3 < 2*, when max{a, /?} < 
2 the authors of pQ prove that the least energy c satisfies 

1 K 

C < — (min{S , (pi),5'(/r 2 )}) 2 

and obtain the existence of nontrivial ground state solution; when max{a, /3} = 
2, the similar results hold provided that the regulator v is large enough. If 
min{a,/3} > 2, the ground state energy is achieved by and only by semi-trivial 
solutions; if min{a,/3} = 2, the similar results hold provided the regulator /i 
small enough. When N = 3 and S^(hi) + S^(h 2 ) < , aZ-^I > they 

obtain the existence of mountain pass solution provided that v is sufficiently 
small. 


For the critical case, that is, a + /3 = 2*, [lj assumed that h(x) is a radial 
function satisfying 

{ h £ L°°(R n ), h>0,h=£0,his continuous in a neighborhood of 0 and 00 , 
h( 0) = lim h(x ) =0. 

|ai|—>-oo 

Then they obtained that if N > 5, max{a, /?} < 2, then (11.611 possesses a nontriv¬ 
ial ground state solution; if TV = 3,4, min{a, /3} > 2, H 2 < Hi < , An -^2 > 

2 

2 N ~ 1 , they obtain the existence of mountain pass solution provided that the 
regulator v is small enough. But for the case of a + /? = 2* and h{x) is not 
radial, they only obtain the existence of ground state solution for max{a, j3J < 2 
and v small enough . 

Note that if 1 < a < 2,1 < /? < 2, a + /3 = 2*, hence 2* < 4, which means 
that the results in |1] do not include the case of dimension N = 3,4 if h(x) 
is not radial. If /i 2 < /ii and /3 < 2, they also obtain the existence of ground 
state solution provided v small enough. However, if fi\ = /i '2 or min{a, /3} > 2, 
whether there still exists a nontrivial solution remains open. At the end of [Tj, 
the authors also impose a list of complicated conditions on h(x) and emphasize 
that if h(x) has a fixed sign, by using the perturbation argument, they obtain 
the existence of nontrivial solutions provided that v is small enough. In [TJ 
Theorem 3.8], they considered the case of a + /3 < 2*, a > 2, /3 > 2. Note that 
for this case there must hold N = 3. More important, in order to prove the 
Palais-Smale compactness condition, they need that 

S%(hi) + S%(h2)<S%. ( 1 . 11 ) 

Under this hypothesis, they obtained a mountain pass solution provided that v 
is small enough. We emphasize that (11.111) can not hold for many ranges of Hi 
and H 2 i for example: Hi = 0 or H 2 = 0, or hi , M 2 > 0 such that Hi + M 2 < j- 

Naturally, we concern the following questions which are still standing open 
before us. 
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1) Whether the role of the parameter v is essential and can be dropped? 

2) What happens if h(x) is not radial? 

3) What happens if h(x) is sign-changing? 

4) What happens if (11.111) is not true, i.e., S%(p i) + S^~(p 2 ) > St? 

5) When a + (3 = 2*, whether there exists a mountain pass solution to (11.11) ? 

The main purpose of the present paper is to study the existence of mountain 
pass solution when the ground state energy is only achieved by semi-trivial 
solution. We will always assume h(x) is sign-changing and not necessary radial. 
Now it is the place to state our results in the current paper. We need one of 
the following two conditions: 

(Hi) h(x) e 

(H^) h(x) is continuous in M. N , h( 0) < 0, limsup/i(:r) < 0. Moreover, we 

| x | —>-oo 

assume that 7 := \\h- ||oo max{a, /3} < 1, where /i_ := min{/i,0}. 

Further, we suppose that h(x) satisfies the following integrable condition: 

(H 2 ) f RN h(x)\z^: 1 \ a \z^ 2 \^dx > 0 for some a > 0 , where zjf is defined in (11.51) . 
Let us denote 


||ft+(aO|| , 2 * ifa + /3<2*, 

" 2* — a — /3 (RiV) 

|| /i-j-(x) 1 1 (j^w) if a-1-/3 = 2*, 


( 1 . 12 ) 


where h + := max{/i, 0}. Without loss of generality, throughout this paper we 
always assume /x 2 < AT. 


1.1 Nonexistence of the Nontrivial Least Energy Solution 

The first main result of the current paper concerns with the nonexistence of the 
ground state to (11.11) for all N > 3 . 

Theorem 1.1. Assume that either /3 > 2, at < AT or cr > 2, /3 > 2, fi 2 = AT = 
[i. Further, suppose that 


I h(x) satisfies (Hi) if a + f3 < 2*; 

( h(x) satisfies (H() if a + fi = 2*. 

Then there exists ©o > 0, depending on N, a, /?, at, AT, such that if 0 < ©o, 
then the least energy of the system is exactly equal to (at)- Moreover, it is 
achieved by and only by 

• (±z\f l ,0) if < AT, where a > 0 ; 
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<7 > 0, Xi G 1 W . 


• (±z£, 0) and (0, ±z£) «/ h 2 = Mi = M ^ 0; 

• (±z CT]Xi ,0) and (0,±^ CTiXi ) */^ 2 = hi = 0, w/iere 


_ ,X — Xi 

z a ,xi = & 2 zi(-),2i(a: 

<7 


[i + N 2 ]^ 


TTiai is, problem EJ ftas no nontrivial least energy solution. 

Remark 1.1. In the above theorem, the constant ©o has an explicit formula 
in terms of N, a, /3, p\, /x 2 . To avoid tedious notations, we prefer to give them 
in Section 4. For the system HM, the authors of m had constructed similar 
results (see[H Theorem 3-4])- But they required that a, (3 > 2 and h[x) > 0. 
Here we improve the results of J7J Theorem 3.4] to the system \1.1\) . When 
hi 7^ h 2 , we only require (3 >2. Moreover, h{x) is allowed to be sign-changing 
in our case. 
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1.2 Mountain pass solution: the case of N = 3. 

In this case, A N = 1. 

Theorem 1.2. Assume N = 3, a > 2,/3 > 2, a + (3 < 2* and h(x ) satisfies (Hi) 
and (R 2 ). Furthermore, assume that 1 < *~ 4 ^ and that 

either S*(«)< S« or 2 («(m) + SM)# > s f (113) 
Assume further 

0< 10- 4 [(2-8/n)§ - (1^4/X 2 ) § ], (1.14) 

f/ien f/ie problem a has a nontrivial weak solution (uq,vo) with uo >0 ,vq> 

0, u 0 v 0 ^ 0. 

Remark 1.2. // /i 2 = jii, then the alternatives of \1.13\) hold true automati¬ 
cally. 

The next theorem is about the case of N = 3 and a + f3 = 2*, which means 
that the coupling terms are of critical. 

Theorem 1.3. Assume N = 3, a > 2,(3 > 2,a + (3 = 2* and h(x) satisfies (H[) 
and (H 2 )• Furthermore, assume that pLi + p .2 ^ 0, | such that either 

s f( m) + sf( m )<sf or 2 ( SW + S(w) )* >s g, (L15) 

Assume further 

0 < min { ^ , K) -4 [(2 - 8 ^ 1)3 - (l - 4/i 2 ) 3 ]|, (1.16) 

i/ien f/ie problem Elf ftas a nontrivial weak solution (uo,Vo) such that uq > 

0, Uq > 0, u 0 uo ^ 0. 


1.3 Mountain pass solution: the case of iV = 4. 

For the case of N = 4, we know 2* = 4 and Aw = 1. If a > 2, /3 > 2, a + (3 = 

2*, we must have a = (3 = 2. Thus, EH) becomes a type of Bose-Einstein 

Condensates (BEC) equation in R 4 : 

{ — Au — Pijf^z = u 3 + 2h(x)v 2 u, in R 4 , 

— Av — P 2 j ^2 = v 3 + 2h(x)u 2 v, in R 4 , (1-17) 

u > 0,u > 0. 

Note that both the cubic terms ( u 3 and v 3 ) and the coupling terms (v 2 u and 
u 2 v) on the right-hand sides of (11.171) are of critical growth. 
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Theorem 1.4. Assume ( H [) and (H 2 ). Suppose + g, 2 0, \ < 2 

such that 

either ( M2 ) + S* ( M i) < S* or 2 ( %)+%) )%gf (1.18) 

Assume further 

16 v ; 
£/ien £/ie problem has a nontrivial weak solution (uo,vq) such that uo > 

0,r o > O,u 0 r 0 ^ 0. 

Remark 1.3. Basically, the upper bounds of 0 in the above theorems are not 
sharp. However, in order to determine an unambiguous range of O, we prefer 
to give the explicit formulas for those constants. The optimal range of 0 is an 
interesting open question. 


Remark 1.4. One of the main difficulties of studying this kind of problems is 
the failure of the (PS) condition due to the critical term |u| 2 and the un¬ 
bounded domain R N , especially for the couple terms |u|° !_2 it|i>|^ and |it| a |i;|^~ 2 i; 
with a + /3 = 2*. One has to overcome the difficulties on determining the com¬ 
pactness threshold. People usually study the case of c < jjS~. One of the main 
innovation of our present work is that we obtain a nontrivial solution with en- 

i N 

ergy higher than jjS~. We also have to overcome the difficulties brought by the 
indefinite sign of the weight function h(x), especially when h(x) is not radial. 


2 Nehari Manifold 

Let D := D 1 ’ 2 (R N ) x D 1 - 2 ^), where D 1 ’ 2 (R N ) is the completion of C(f(R N ) 
with respect to the norm 


'■=( \Vu(x)\ 2 dxj 

v Jr n j 


1/2 


By the Hardy inequality, when 0 < p < ^ , IMId^/rw) is equivalent to the 

following norm: 


I NU“(/ (|Vn(x)| 2 -^)dx) 
W R« \A > 


1/2 


For simplicity, we will also use the notation of ||u||o to represent ||w||.Di,2(RiV). 
For (u, v ) € D, define the norm 


H(«,«)l| D =(lK 1 + IHI 


1/2 
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A pair of function (u, v) is said to be a weak solution of problem (11.11) iff 


/ Vit • S7<pidx — fii / ^^-dx + / S7v ■ S7if2dx — ^2 / -rrrdx 
Jr n Jr n Fl Jr n Jr n Fl 

— / |w| 2 utpidx — / |t;| 2 ~ 2 v<p 2 dx — a h(x)\u\ a ~ 2 u\v\^ ipidx 

Jr n Jr n Jr n 

— /3 f h(x)\u\ a \v\ fj - 2 vip 2 dx = 0 for all ^2) G D. (2.1) 

Thus, the corresponding energy functional of problem m is defined by 

= ^II(wf)||d - -jz (|MIl2*(rjv) + ||v||f2* (RW) ) - h{x)\u\ a \vf dx 

(2.2) 

for all ( u , v) £ ID. The associated Nehari manifold is defined as 
A f := j(n,u) £ O\{(0, 0)} : J(u,v) = oj, 

where 

J{u,v) = ($'(u,v),(u,v)} = ||(w,u)||^- (|M|£*2 . (rJ V) 

+ll u llia*(RW ) ) - (a + 0) J h(x)\u\ a \vfdx, (2.3) 

and <&'(u,v) denotes the Frechet derivative of $ at (u,v), (•,•) is the duality 
product between D and its dual space D*. We have the following properties on 
the Nehari manifold. 

Lemma 2.1. Assume a + (3 < 2*. In particular, if a + /3 = 2*, we require (H[) 
instead of (Hi). Then V(u, v) £ 1D)\{(0, 0)}, there exists a unique t = L Ui „) > 0 
such that t(u , v) = ( tu , tv) £ N. Furthermore, there exists 6 > 0 such that 
t(u,v) > $ for all (u,v) £ S := |(u,u) £ ID : ||(u, u)||n = l|, and Af is closed 
and bounded away from (0,0). 

Proof. For (u, v) £ D\{(0, 0)}, we denote that 

(a := |M| 2 2 *( R n) + IMIl 2 *( R n) > 0; 

< b := (a + /3) f RN h(x)\u\ a \v\^dx\ (2.4) 

U : = IIWfOIId > o. 

Then, ^4 )(tu,tv) = —tg(t), where g(t) := a t 2 ~ 2 + b t a+ — c. Firstly we 
consider the case b > 0. Note that there exists a unique t 0 > 0 such that 
g(to) = 0. Moreover, g(t) < 0 for 0 < t < to and g(t) > 0 for t > to- Secondly 
we consider the case fe < 0. If a + /3 < 2*, there exists some s > 0 such that 
g(s) = 0. Let to be the minimum of the solutions of g(t) = 0, that is, <?(fo) = 0 
and g(t) < 0 for t < to- For V t > to, it is easy to check that g'(t) > 0. Thus, 
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g(t) > 0 for t > to. Then, to is the unique solution to g(t) = 0. If a + /3 = 2*, 
by the Young’s inequality and (H[), we have 

|b| < 12* [ h-(x)\u\ a \vfdx\ 

Ji» 

— ^ II /i -IIoo(^II' u IIl2*( R jv) + ^rlMI L 2* (R iV)) 

< y/i-Hoo max{a, /3}a 

< a. (2.5) 


Hence, a+b > 0. Then there exists a unique positive solution to to the equation: 
g(t) = (a + b)t 2 ~ 2 — c = 0. In particular, g{t) < 0 for 0 < t < t 0 and g(t) > 0 
for t > to- Let tr u>v \ := to be dehned as above, we finally obtain that 


> 0 

for 0 < t < t( u 

< 0 

for t > 

'( U,V ) 

> 0 such that | 


— $(tu,tv) = - tg(t ) 


ma x$(tu,tv) and t^ u ^{u,v) G M. For u = ( u,v ) € S, since h( x) (R^) D 
L°°(R n ) and a + /3 < 2*, there exists some C > 0 such that 


a = I ML 2 * (R w ) + IM Il 2 *( R jv) < C\ 

|6| = \(a + (3) f RN h(x)\u\ a \v\Pdx\ < C; 

c = 1. 


We consider the equation g(t) = a t 2 2 + b t a+l3 2 — 1 = 0. If b < 0, we have 
at 2 _2 > 1, hence t > ( a)~ 2 *- 2 > (C)~ 2 *- 2 . If b > 0, then we have either 

at 2 ~ 2 > L or bt a+l 3-2 > Thus, either t 2 or t > ■ 

Therefore, there exists <5 > 0 such that > 6 for all (u, v) £ S. Therefore, 
A f is bounded away from (0, 0). Obviously, AT is closed. □ 


3 Analysis of the Palais-Smale Sequences 


In this section, we perform a careful analysis of the behavior of the Palais-Smale 
sequences with the aid of the concentration-compactness principle in mm , 
which allows to recover the compactness below some critical threshold. Set 


CN,a,/3 '■ = 


1 4max{/ii,/i2} 

/ (N- 2) 2 


- 1 


max{a, /?} 


(3.1) 
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Lemma 3.1. Assume P 2 = 0, p\ = p G (0, 


( N 4 2 ^ ) and 


either 


(Hi) 

a + /3 < 2* 


or 


(H'i) 

a + /3 = 2*' 


Let {(u n ,v n )} C J\f be a Palais-Smale sequence for $|_v at level c < u). 

Then, there exists some constant C, such that ||(it n ,u n )||B < C for all n G N 
and $'(«„, v n ) —> 0 in the dual space B*. Moreover, 


(1) for the case of 
subsequence; 


(Hi) 

a + /3 <2* 


we have (u n ,v n ) —► (uq,vq) in B up to a 


(2) for the case of 


if h{x) is radial, we have (u n ,v n ) 


(HO 

a + /3 = 2* 

(uq,Vo) in B up to a subsequence. However, if h(x) is not radial, we 
obtain the same result provided that the additional hypothesis 0 < Cnf a a 
(see \3.1\ ) ) holds. 


Proof. The ideas for proving this lemma are quite similar to the cases of pi, P 2 > 
0 and pi = P 2 = 0 in [35] Lemma 6.3], we omit the details. □ 


In the next section, we will study the nonexistence of nontrivial ground state 
solutions. In view of the nonexistence of the ground state to the system, we will 
investigate the existence of mountain pass solutions of system m- For this 
goal, we need an improved Palais-Smail condition at higher energy level. Let 


I u( u ) 



u G D 1,2 (R n ). 


(3.2) 


We consider the following modified problem to find the nonnegative mountain 
pass solutions to m, 


—Am — Pij^i = u+ 1 + ah(x)u+ 1 v^_, 

-Am - P 2 f^p? = v 2 + -1 + /3h{x)u‘fv l l^ 1 , 

where u+ = nraxjzi, 0}. The weak solutions to problem (13.3[l are critical points 
of the following functional $ : B — > R given by 



$(m,m) = / Ml (u) + / M2 (m) 


h(a 


*+ </ + 


dx. 


(3.4) 


where 


Im H 




* = 1 , 2 , 


(3.5) 
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and 


(3.6) 


QJw) = / \\7w\ 2 dx — p / r~^dx. 

J i« J M 

Obviously, the critical points of $ provide nonnegative solutions to the original 
problem ED- We denote by A f the Nehari manifold associated to <f>, i.e., 

A 7 = |(u,w) S D\{(0,0)} : {®'(u,v), (u,v)) = oj. (3.7) 

Assume N = 3, \ © < min{Ci, C 2 }, where O is defined in (11.121) and 


(l-4^)§-(A) § (l-4 M2 ) § 
(i)^(l - ’ 


(3.8) 


U 2 := 


2 — y/2 

8-/3 " " 4q + /3-6 “ “ 8-2 J 7 ‘ 

2 b (1 — 4 ^ 2 ) 6 (3S 4 + a 1 


(3.9) 


^ T 2 * 

It is easy to check that C 2 > 5 x 10~ 4 , C\ > 10 -3 [(1 — 4/n) 3 — ( ^ ) 3 (l~ 4/i2) : 


Lemma 3.2. Assume N = 3, a, /3 > 2, a + /? < 2* and | < j 5^77- Aef 
{(wnj'Cn)} C 77 be a Palais-Smale sequence for <E>|^ at level c£l. Then, there 
exists C > 0 such that ||(Tt„, v n )||o < C for all n £ N and $ ( u n ,v„ ) —> 0 in the 
dual space B*. Furthermore, if O < min{Ci,(72} and c satisfies 


1 jv / \ 1 „jv , , 

— 5= (p 2 ) < c < — S 2 (/z 2 ) + 


inf v); 

(u,v)£Af 


(3.10) 


c ^ —S' 2 (p 1) and c -^-S 2 for all l £ N\{0}, (3.11) 

f/ien up to a subsequence, ( u n ,v n ) —»• (u o,vq) in D. 


Remark 3.1. In [71 Lemma 3.5], the authors only considered the case S 2 (/ai)+ 
AS^(^ 2 ) < ^S^. However, if p 1 + p 2 < f/ien ^S^(^1) + ^S^ (/z 2 ) < 
-j^S" 2 " will never meet. In particular, the sign-changing h[x) makes the proof in 
Lemma 13.31 more complicated. 


Proof. We divide the proof into five steps. 

Step 1: It is easy to show that {( u n ,v n )} is bounded in D and $ (u n ,v n ) —t 0 
in the dual space D*. Up to a subsequence, {(u n , n„)} ng N converges weakly to 
some (uo,vo). Hence, ((u n )~, (v n )~) —> (0,0) strongly in B. It follows that 
(k) +, (n n ) + ) is a bounded Palais-Smale sequence of <F. For ((zt„)+, (v„)+), 
we may find a t n > 0 such that t n ((un)+,{v n )+) £ AfdM. Since (u n ,v n ) £ 
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A1 and ((«„)_,(%)_) —► 0 in HD, we have t n 1. Hence without loss of 
generality, we can assume that u n > 0,v n > 0, {(u n , u n )} ragN C AT D Af is a 
Palais-Smale sequence for $ at level c. Notice that <f>(w n , t>„) = ^>(u n ,v n ). For 
the simplicity, we use ||u||o to stand for ||u||£)i, 2 ( R N). There exists ( uq , vo ) £ D 
and a subsequence, still denoted as {(tin,^)} N such that 

{u n ,v n ) ->• (u 0 ,v 0 ) weakly in D, (u n , v n ) -7 (u 0 , v 0 ) a.e. intt w , (3.12) 

and (u n , v n ) -» {u 0 , v 0 ) strongly in L7 oc ( rjY ) x for all 7 £ [1,2*). 

(3.13) 

In view of the concentration-compactness principle due to Lions [T9], [20], there 
exists a subsequence, still denoted as { (u n , ' y n)} n6N , two at most countable sets 
J and /C, set of points {xj £ K JV \{0} : j £ and {//& £ RAr \{0} : k £ /C}, 
real numbers CiiPjiJ € J7, CfciPfc)^ £ /C,Co, Po, Co and Po suc h that 


'|Vu n | 2 ->• dp > |Vu 0 | 2 + X] C j6 Xj + Co^o, 

iej_ 

|Vu n | 2 ^dp> |Vu 0 | 2 + E CAk + Co<*o, 
fce/c 


dp = |u 0 | 2 + J2 Pjdxj + Po^o, 


jeJ 


12* 


(3.14) 


dp = M + E Pk S yk + Po S 0, 


fce/c 


^ ^ de - + e 0 s 0 , 

w ^ d0 = |7j2 + 9q5o- 


Define 


Coo := lim limsup 


n->00 J\x\>R 


\X7u n \ 2 dx, poo := lim limsup 


Coo '•= lim limsup / \S7v n \ 2 dx, p^ := lim limsup 
R—>oo n—toQ J\x\>R R—>oo n—>-oo 


00 n—>oo J\x\>R 

f 

\x\>R 


dx, 


v n \ 2 dx. 


It follows that 


Coo := lim limsup 


R^fOO n—toc J\x\>R 


|V(u„ - u 0 )| 2 da;, 


Poo := lim limsup / \u n — uo\ 2 dx, 
_R->°o n _> 00 J\x\>R 


Coo := lim limsup / 

R-^roo n—too J\x\>R 

Poo, := lim limsup / 

fl->°o 00 J\x\>R 


|V(u n - V 0 )\ 2 dx, 


|v„ - u 0 | 2 dx. 


From the Sobolev’s inequality, it follows easily that 

Spf < Cj for all j & J] S~pl* < C fc for all k £ 1C. 


(3.15) 

(3.16) 

(3.17) 

(3.18) 

(3.19) 
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Step 2: We prove that either u n —> uo strongly in L 2 (R w ) or v n —> vo strongly 
in L 2 (K^). If not, then there exist some jo € J U {0, 00 } and ko £ JC U {0, 00 } 
such that pj 0 > 0 ,p k > 0. Since a + /? < 2*, we have p 0 > S^(pi),Pk 0 > 
(p 2 )- In order to make the present paper easy to follow, we prefer to give 
part of the proofs. Indeed, for e > 0, let cj>^ be a smooth cut-off function centered 
at Xj , 0 < <pj < 1, such that 


m = 


1 if \x — Xj\ < | 
0 if \x — Xj\ > e 


and | y y <j>j(y + Xj)\ < - for all y = x — Xj £ 


Testing <&'(u n ,v n ) with (u n </>j, 0), we obtain 

0 = lim ($'(u„,v n ), (u n $,0)) 

VI _' J ' 


(3.20) 


= lim 

n—>■ 00 


/ ( r 

J R w V 


Vrt n | 2 ^ + u n Vu n ■ \ur, 


- ah (x) I u n I “ I v n 1 13 (f)fj dx. 
Notice that for all e > 0 fixed, 


(3.21) 


/ R N 


u„V«„ • S7(pjdx 


| <e 


U n Vu n ■ S7(fijdx 


< | x—Xj | <e 


(lln - Wo)Vw„ • X74>jdx 


uoS7u n ■ 'V(j) £ i dx 


' § <|ai —Xj |<£ 

:= / + II. 


(3.22) 


(3.23) 


Note |V^| < |. Without loss of generality, we may assume that 

(u„, v n ) -► (u 0 ,v 0 ) strongly in x L 7 OC 0^ N ) for a11 7 € [1,2*). 

Then by (13.2311 and the boundedness of S7u n in L 2 (R N ), we have 

I —> 0 asn-> + 00 . (3.24) 

and that u n — i uq 


Note that II can be taken as a linear functional in L 2 
in D 1 ’ 2 )®^), we have 


II 


\<\x-Xj\<e 


uoVuq ■ Vcffjdx as n —> + 00 . 


(3.25) 


Since (u n ,v n ) — i (uo,vo) weakly in IB), ( uq,vo ) is a weakly solution to problem 
COD- Taking (u o <^,0) as the testing function, then we have 

0 = / [|Vit 0 | 2 <M +m 0 Vm 0 • - </>?K| 2 * ^ ah{x)\uo\ a \vo\ 0 (t>f\dx. (3.26) 

Jr n 
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Further, 


lm N 


\u n \ a \v n f (jf.dx = 


\u n \ a \v n \ p (jfidx 


'\x—xj\<e 


< 


u n \v n rax 


l\x-Xj\<S 


fdx 


< ( 


' \x—Xj\<e 


dx)' 


\x-xj\<e 


\v n \ 2 dx) '■ 


1 dx) 


J\x-Xj\<£ 

0(£ 2* ). 


Since h(x) £ L°°(R Ar ), we have that 


ah(x)|u„|“|'u n | / V)dx = 0(e “). 


Jr n 


Especially, 


m N 


ah(x)|ito|“|'co| /3 0^a ; = 0(e 2 * ^ ' 


By (13.141) and (13.21IW(13.28[) . let e —> 0 we obtain that 

H - Pj < °- 


(3.27) 


(3.28) 


(3.29) 


By (13.191) . we conclude that for all j £ J, either pj = 0 or pj > S%, which 
also implies that J is finite. For the details about the similar results related 
to j £ {0, oo} and p k , k £ K, U {0, oo}, we refer the readers to .35, Lemma 3.2]. 
Then we have 

c = <f>{u n ,v n ) + o( 1) > (i - —^—j)(S{p 1 )p]^ +S(p 2 ypT Q ) 

+ ( a + /3 ~ ^)(Pjo +Pk 0 ) 

> j-f(S^(vi) + S%(n2)), 

which is a contradiction with (13.101) . 

Step 3: We prove that either u n —> uo strongly in D 1,2 )!^) or v n —> vq strongly 
in D 1,2 (R n ). By Step 2, if u n —>• uq strongly in L 2 (l w ), then 


II U n - Uoll^ = (<&'(u n ,v n ), (u n - u o ,0)} + o(l) = o(l) as n ->• +oo. 


Hence, u n —► u 0 strongly in D 1,2 (M. N ) in this case. If v n —► v 0 strongly in 
L 2 (R^), correspondingly we have v n —»• Vq strongly in D 1,2 (WL N ). 
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Step 4: If v n —> vq strongly in Z3 1,2 (R Ar ), we prove u n uq strongly in 
D 1,2 (M. N ). We argue by contradiction and assume that u n —^ uq weakly but 
none of its subsequence converges strongly to uq. 

Firstly we claim vq jk. 0. If not, vo =0, it is easy to check that {u n } is a 
nonnegative Palais-Smale sequence for the functional dehned in (13.21) . at the 
energy level c = lim 1^ (u n ), which can be calculated as following: 

n—> oo 

c = lim $(u n ,v n ) 

n—too 

= r |™ o 2 IK U "’~ 2 *" (r-^) + l^n| L 2 *( R N)) ~ J^ h(x)\u n \ \v n \^dx 

— lim In 1 (un) T k/i -2 (^n) / h(x)|u n | | v n | dx 

n—too J RN 

= lim Lj ( u n ) since v n —> vq = 0. 

n—>■ oo 

Then the result of [26l Theorem 3.1] (we take K(x) = 1, A = /xi in [26]) implies 
that there exists some to, l € N such that 

c = lim$(u„,u„)= lim / Ml (u„) = / Ml (u 0 ) + 

n—> oo n—t oo 

If uq = 0, then by (13.1 111 , we have to yk 0,1 ^ 0. In this case, c > jjS% + 
{Hi), a contradiction with (13.101) . If uq jk 0, since uo > 0, we have uo = zg 1 
for some a > 0 and f RN |m 0 | 2 *^ = S% Z Ml (u 0 ) = (/ii). By (13.111) , we 

obtain to ^ 0. Then c > jjS~z + jfS ~also a contradiction with (13.101) . 
Thereby the claim uo ^ 0 is proved. 

Thus we may assume that u n — k uq weakly but not strongly in D 1,2 ( M. N ) 
and v n —> vq ^ 0 strongly in H 1,2 (R Ar ).If uo = 0, then vq weakly solves 

By the known result in Section 1, we have vq = z£ 2 which is defined in (11.51) for 
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some a > 0. Thus, |uq| 


L 2 * (R N ) 


= INIlL = -S' 2 (mi)- Therefore, 




>(i-— 

~\2 a + /3 


II('uo,'l>o)IIb + b 0+ ^ Cfc 

jE^TU-fC^oo} fcG/CU{0,oo} 


\L 2 *(R n ) 

e 


l Vn li 2 *(R iv ) 


)+o( 1) 


( 


1 


1 


a + /3 2 

1 1 


(|rt 0 | 2 + \v 0 \ 2 )dx + E p 7 -+ E p fc 
R iv iej'u{o,c»} J fce/cu{o,oo} *■. 


+ 


a + /3 

(— -— 

Va + /3 2* 


(a + /3 2* 


E 

je J'u{o,oo} 


L 2 *( R N ) 

1 1 


0 ] + (^9 " F) [ 


+ (^jTg “ + 5 (w)^o* + S(mi)p£ ] 


+ ft 


) [ .J^ ft + Po + Poo] + o(l). 


>(1) 


E pj] + o(l) 

j&JU{ 0,oo} Ji 


Since u n —>■ zzo weakly but not strongly in Z) 1 ’ 2 ^^), there exists some j G 
J" U {0, oo} such that pj ^ 0 and that 


c + (y^s - b sfa) ft + (^b “ f) 


Pj 


1 • .2* , (. 1 1 


-7vl l, °l^ 2 *(R N ) + Va + P ~ 2 


)'S'(Mi) • S ' 2 '^(Pi) + ( 


a + f3 2- 




1 


=^(S 2 (pi) + <5 2 (M2)), 


a contradiction with (13.101) . Hence we can assume that uq ^ 0. It is clear that 
(uq, vo) € AT D J\f and 


$(u 0 ,v 0 ) = 


N 


zig dx + [ Vq dx) + a ^ —- 

J R N ' 2 


Ir n 


h(x)uQVQdx. (3.30) 


Since v n -» vq in D 1 ' 2 ( R w ) and u„ —^ uq weakly in D 1,2 
that 


we can obtain 


lim [ h(x)u^v^dx = [ h{x)uQVn.dx. (3.31) 

n->oo J RN J rN 


Combining these facts, we have 


$ [u-n ,v n ) — 2 II ( u n J V n ) 11B 


2* (l' u,l lL 2 *(M iV ) l Vn li 2 *(R iV )) J N h(x)\u n \ | v n \Pda 


1 

Z N 


[ U n dx + ^7 [ v n dx ft ———^- 1 / h(x)u°VPdx, 

J RJv IV Jrjv 2 


/R^ 
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(3.32) 
























(3.33) 


lim [ v%dx = [ v\fdx, 

ra->°o J rN J r jv 

lim / uf l dx= Uq dx -t- pj + po + Poo • 
n ^°° J E» Jr n ~Tlr 


(3.34) 


j£J 


By (13.301) ~ (13.341) . we obtain that 


vq) = lim $(u n ,v n ) - — ( V' pj + po +Poo) 

n.—¥no K z ' 


jej 


= c ~ x(J2Pi + P° + P°°) 


j£J 


< C--S*(p!) 

< ^(p 2 ). 

Note that 

Mg dx + a / h(x)uQVQdx = HwollL > S'(pi) 


m N 


Ir n 


, 2 * 


R N 


dx ) : 


(3.35) 


(3.36) 


h(x)uQ r VQdx < 0, then / Uq dx > S 2 (pi). Similarly, we have 


If 

J i» 

S~^ (p 2 ). Thus, 


/R« 


lR N 


Vq dx > 


*(u 0 ,v o) = (i - ^)ii(«o,«d)ns+- ^)(ii«oin;. + i^oiii;.) 
> (J - 7n-p)5(Mi)ll«o||| a - + d - -^)5(p 2 )|M| 2 L2 . 


a + p 


ot + [3 


+ ( ^ - + IMS-) 


(3.37) 


a contradiction with (13.351) . 


If 


h(x)uQVQdx > 0, then by (13.301) and (13.351) we deduce that 


L 2 *(RW) + II v 011^2* ( R JV) < S' 2 (p2). 


Similar to (13.361) . we have 
Vq dx + P / h{x) 

J RN 


/R N 


Ug Vq dx = 


> S(p, 


R w 


(3.38) 


* dx)^. (3.39) 
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Then by (13.381) . (13.361) and (13.391) we obtain that 


+ P@S N + 2a (iJ,2)\\vo\f L 2* {kN) (3.40) 

and 

' S '(Mi)II u o||£2* (r jv ) < llwoll^^RiV) TaQS 1 4 ' 3 (M2 )||wo||^2*( R jv). (3.41) 

Since N = 3, we have S(n) = (1 — 4 n)%S, then (|3.40|) and (13.411) are equivalent 
to 2 2 

(1-4 M2 ) § S< ||uo|lla (R 3 ) +/30(l-4 M2 )^^||uo||^ ( 2 R3) (3.42) 

and 

2 2/3 

(l-4^ys< ll^ollie^S) +«©(1 -4/U2) 3 ^IKIISTja). (3-43) 

Note that a > 2 and that /(f) := ti + a0(l — 4^ 2 )§ /3 5 /3 t a ^ is increasing in 
(0, +oo). If 0 < Ci, where Ci is given in (|3.8p , then 

/(±(1-4/x 2 )S*) 

= [|(1 - 4 p 2 )Si ] 1 + 00(1 - 4/x 2 )W [^(1 - 4/z 2 )Si] ^ 

= (i) § (1 - 4/i 2 ) § 5 + (i) ^00(1 - 4 M2 ) §/3 (1 - 4 M2 ) ^ S p+3 ^ 

< (l-4/x 1 ) f 5. 

Recalling (13.431) . it follows that ||«o||^ 2 -( R jv) > |(1 — 4/r 2 )S'i = (/r 2 ). Sim¬ 

ilarly, we have ||t>o||f 2 . (RJV) > \S%(n 2 ). Thus, ||u 0 ||| 2 . (RW) + |Mlf 2 * (RW) > 
a contradiction to (13.381) . 

Step 5: If u n —» u o strongly in Z? 1,2 (IR^), we show that v n converges strongly. 
For this case, K, U {0, oo} is reduced to be at most one point. In fact, if not, 
by (3.31), (3.39), we obtain that c > -^5^(/x 2 ) > -^(S^(/xi) + 5^(/z 2 )), con¬ 
tradicting to (13.101) . Assume that v n —‘ vo weakly but none of its subsequence 
converges strongly to vq. We claim rto ^ 0. If not, uo = 0 and vq ^ 0, then 
vq £ D 1,2 (R n ) is a weak solution to 




- Av 0 - M2y-p7 = M 


12 — 1 


V 0 . 


(3.44) 


Denote 


E\ := inf {/ M2 (u)|u ^ 0 is a solution of problem (13.441) }. 

E 2 := inf { / \v\ 2 dx\v ^ 0 is a solution of problem (13.441) }. 

Jr n 
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where J M (v) is defined in (13.21) . Then it is well known that E\ = jjS? (^ 2 ), ^2 = 
(^ 2 ), and they are only achieved by 


c > 


( ±z^,a>0 if/u 2 >0 

[ ±Zcr tXi ,<7 > 0 ,Xi <E if /r 2 = 0. 

Thus, by the fact that {v n } concentrate at exactly one point, we obtain that 

J^(J rN v o* dx + S ^(^) - ~ + 

contradicting (13.101) . On the other hand, if uo = 0, vq = 0, then v n solves 

— Av n — H 2 T~j 2 ~ v n 1 = o(l), in the dual space(U 1 , 2 (K 7V ))*. (3.45) 

FI 

Then an analogous argument as that in step 4 will lead to that 

C = &(u n , V n ) + o(l) = 1 ^ 2 ( V n ) + O(l) —* / M2 (r>o) + —S 2 (/i 2 ) + — *5 2 
= (m 2) + for some m,leN 

1 jy -1 iV 

as n —► 00 . By (13.1 ID . we have m / 0,Z 7 ^ 0, then c > -^5 '2 (^ 2 ) + j^S'" 2 ", a 
contradiction with (13.101) . Thereby the claim uo ^ 0 is proved. Thus we may 
assume that u n —> uo ^ 0 strongly in D 1>2 (M. N ), and v n —*■ uo- If uo = 0, then 
uo weakly solves 

Auq 1^1 "j ^ 2 " ^0 0 ? 

FI 

hence uo = zg 1 for some a > 0 and / |rto | 2 dx = S~*(ni). When /r 2 > 0 

Jr n 

(similarly for the case of /z 2 = 0 ), by the fact that {v n } concentrates to exactly 
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one point, we deduce that 
1 1 
^2 _ oT+ (3 


c = 


t 


(1 - ^r?) [»“»e. + »*’»» 

+ XX fc + (Co ~ P2#o) + (Coo ~ M2^oo) + ^ - ^r) 

(M“ + |t'o| 2 )dx + + Po + Poo + °(1) 


fce/c 


L JR W 


fce/c 


> 


a + /? 


1 


> 


1 

a + /3 2* 

1 


(5 

+( 
fi 

V 2 a + ,0 

+ (aTd f) 0 u °^* + /\) for some * e /cu { 0 , 00 } + 0 ( 1 ) 

(3.46) 


) IKII^ +Sj2 p k + S '(^2)(Po* + p™ ) 

fce/c 

1^0 || 2* + + A) + Poo) + °( 1 ) 

keic 

)(5(/ii)||u 0 |||. + S(p 2 )p] 


> (Pi) + S* (M 2 )) +o(l), 

a contradiction with (13.101) . Hence, both uq ^ 0 and r>o ^ 0. Hence, 


c = $(u„,u„) - (u n ,v n )) + o(l) 

= / < cfe + 


IV 


(/ u n dx+ f vl*dx)+— ^ — -f , 
V 1r» Jr" 7 2 JR W 


a + /3-2 


(3.47) 
h(x)u"v^dx + o(l), 

h(x)uQVQdx. (3.48) 


it follows that, for some k € /C U {0, 00 }, 

c = 4 ( [ u 2 0 dx+ [ Vq dx + p k ) + 

^ Jl» JRW 

On the other hand, recall that (^(un,, u n ), ( uo,vq )) = o(l), we have 

||(m 0 ,'*4>)||b = / Uo*da;+ / ujfdx + (a +/3) / h(x)u%v%dx, 

Jr n Jr n Jr n 

i.e., (u 0 ,vo) £ AC If p 2 > 0, by (3.31), (13.481) and assumption (13.101) . we obtain 
that 

$(m 0 ,co) = / Uq *dx + f v$*dx) + —-[ 

7V JR» 7 2 Jr/ 

= c-|<l^( / , 2 )+ inf Mu,v)-% 

iV iV (u,u)GA/^ iV 

- 4 ' s '^(p 2 )+ inf $(u,u)- -^-5 , ^(/i2) 

JV (u,v)gAT iV 

= inf $(u,ti), 

(u,v)SlM 

a contradiction with (w 0 ,^ 0 ) £ AC. 


h(x)uQVQdx 


(3.49) 

□ 
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Next, we consider the case of a + /3 = 2*. For this case, we can not expect 
Pj = 0 or pj > S~ 2 for j ^ {0,oo} any more. Because the Holder’s inequality is 
not enough to ensure 


h(x)\i 


f&dx —^ 0 uniformly as £ —> 0, 


(3.50) 


where <ff^ is defined in (13.201) . Thus, the step 1 of Lemma 1 3.2 1 fails for the case of 
a + (3 = 2*. We will impose more conditions on h(x) to overcome this difficulty. 
Assume pi + p 2 7 ^ 0, take £i small enough such that 


. . N ~ 2 / 
2(1 - ei )-* T - > (1 


4 ^1 

(TV- 2) 2 ' 1 (TV- 2) 2 ' 


(3.51) 


For example, if TV = 3, we choose E\ satisfying 


ei < 1 - [l - 2(p 1 +p 2 )\ 3 


and if TV = 4, we may take e i satisfying 


£i < 1 


[(1- px) 2 +(1- p 2 f 2 ] 2 

4 


(3.52) 


(3.53) 


Lemma 3.3. Assume a + /3 = 2*, 0 < p 2 < pi < 1 — 4/x 2 < 2 — 8p\,a, f) > 

2, pi+ p 2 ^ 0. Let {(u n , u ra )} C TV" be a Palais-Smale sequence for at level 
cel. Then, there exists some constant C such that ||(u n ,u n )||D < C for all 
n £ N and (u n ,v n ) —7 0 in the dual space D*. Moreover, if c satisfies 13.1(A) . 
ESP and 


0 < min 


{ 


1 - (1 - E-i)n 

3 . .a-21 

22 a(l — Ei)^~ 


1 - (1 -£i)« 

2§/3(1 - Ei)^ 2 


C-y,C 2 


}• 


(3.54) 


where E\ satisfies \3.52\) and C \, C 2 are defined in m and m , then, up to 
a subsequence, (u n ,v n ) —7 (u o,Wo) in D. 

Remark 3.2. Under the assumptions of Lemma \3.!A we can have 


. f l-(l-£i)» 1 - (1 -£i) 

nn < —g - 

*• 2 = a(l — Ei)^- 2f ^(1 - £i)~2 
Pi + P-2 - (Pi + P2) 2 


—,C 1 ,C 2 \> 


} 


• J' 

nn < - 


, 10~ 3 [(1 - 4 Ml )f - (i)^(l-4 M2 ) f ],5x 10- 4 }. 


Proof. We need several steps. 

Step 1: There exist an at most countable set J (for simplicity, here we view 
J as the set J U AT in Lemma 13.2ft , the set of points { Xj £ R^jO} : j £ j}, 
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real numbers Q,pj,(j, Pj,j £ J, £o, Po, Co an d An such that 

'|Vu„| 2 ->• dp> |Vzt 0 | 2 + X] +Co<5o, 

jej_ 

|Vw„| 2 ->■ dfi> |Vz'o| 2 + X] C jS Xj + CqSo, 

jej 

|m„| 2 * dp= |u 0 | 2 * + E PjSxi + PoSo, 

< jeJ (3.55) 

K| 2 * dp= |v 0 | 2 * + E Pjtixj +P 0 S 0 , 

jej 

d0 = + 0 o<5o, 

, \ dd = |^2 + ^o<5o. 

Note $'(u n ,v n ) -J 0 in D* and {( u n ,v n )} is bounded, we obtain that 
0 = lim (&(u n ,v n ), (u n <f> e j,0)) 

n—> oo J 

= lim / ( \\/u n \ 2 (j) e j + • Vcjfj — Pi-f-w- 

n—too j rN \ | a; | 


-^|m„| 2 * - ah(x)\u n \ a \v n f(j)fjdx. (3.56) 

By (H[), we can follow the process of [35] Lemma 3.2] and obtain that 

po = 0 or po > S%(pi), Poo = 0 or poo > 5"^(pi). (3.57) 

Similarly we can obtain that 

N N 

Po = 0 or po > S^{p 2 ), Poo =0 or poo >5^ (/x 2 ). (3.58) 

For Xj £ R Ar \{0}, if h(xj) < 0, then we can argue as above and obtain that 

Pj = 0 or pj > S %; Pj =0 or p,- > . (3.59) 


Next we consider Xj £ R JV \{0} with h(xj ) > 0. Then one of the following holds: 
(!) Pj = Pj = 

(2) pj = 0 and pj > 0; 

(3) pj > 0 and Jjj = 0; 

(4) pj > 0 and pj > 0. 

If (3) holds, then (13.501) satisfies. By (13.561) and Xj ^ 0, we can obtain Cj~Pj < 0. 

N 

Then the Sobolev’s inequality implies that either pj = 0 or pj > S~. Thus, if 
(3) holds, we have pj > S~. Similarly, if (2) holds, we have pj > S~. If (4) 
holds, we can not apply (13.501) but we claim 

S(pf +pf)>S%(p,i) + S%(p, 2 ). (3.60) 
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2* N 

Without loss of generality, we can assume that max{ Pj,Pj} < 2~ S. Then by 
the Sobolev’s inequality and (13.561) . we have 



aepf(2%-S%)& 


< 0 , 


(3.61) 


which is equivalent to S' < 



f(t) = t 1 ~^+aG2%S 


(N-2)0 

4 


a-2 

t~2^~ 


_ _ 0 _(TV- 2)0 a Z 2 _ 

+ O02 2 S J p 2 . Consider the function 
, which is increasing in ( 0 , +oo) because of a > 


2. If 0 < 


1 - (1 -£i)w 

2^a(l — £i)t* 


then we can compute /((1 — £i)S”) as following: 


/((1 - £i)s”) = [(1 - £i)^ + 0022 (1 _ < S, 


(3.62) 


which implies that pj > (1 — £i)S 2 and SpY > (1 — £ 1 ) N S 2 . Similarly, if 


0 < 
have 


1 — (1 — £-\) N JL , . N — 2 N ___ 

-we have Sp 2 > (1 — £ 1 ) N S 2 . Hence, by (13.6211 we 


22/3(1 - £i)t* 
S(P] 


-pj) > 2 (l- ei )^ST >S^(pi)+S^(p 2 ). 


Step 2: We prove that either u n —> u 0 or v n —> v 0 strongly in L 2 (R N ). If 
not, then there exist some jo £ J U {0} U { 00 } and ji G J U {0} U { 00 } such 
that pj 0 > 0, Pj x > 0. If jo 7 ^ ji, then we have pj 0 > S~ (pi),Pj 1 > S~ (p 2 ) and 
obtain that 


c = $(«„,*)„)+o(l) 

= ^||(Un,«„)||£ + 0 (l) 

> jfS(jn) P % + jjS(p 2 )pY 

> ^(S^M+S^(/x 2 )), 

which contradicts with (13.101) . If jo = ji = j and j € {0,oo}, then we have 

N N \ / N N v 

Pj > S' 2 {pi),Pj > S" 2 " (p 2 ) and c > — (S" 2 " (/r^TS" 2 " (^ 2 )), also a contradiction 
with (13.101) . If jo = ji = j and j ^ {0, 00 }, recall (13.541) and (13.601) we see that 

c>-^S(pf +pf r ) > -^(S^(/ii) + St(/j 2 )), 

also a contradiction with (13.101) . Thus, we have either u n —> u 0 or v n —► Vq 
strongly in L 2 *(R JV ). 


1 / TV 

Step 3: By the above arguments, under the conditions of c < — [S~(p\) + 

S" 2 " (/r 2 )) and (13.541) . we obtain that either pj or p^ equals 0 for any j ^ { 0 , 00 }. 
Then (13.501) is satisfied. Further, 


TV TV TV TV 

Po > s^ (pi),Poo > S^(pi),p 0 > s^(p 2 ),p 00 > s^(p 2 ) 


(3.63) 
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and J is finite. For j & J we have either 


(pj,Pj) = (0,0) or (pj = 0, Pj > S*) or ( Pj > S* lPj = 0). (3.64) 

Thus, the steps 3 ~ 5 in Lemma l3.2l are valid here, and we finish the proof. □ 

Consider N = 4, a = /3 = 2, (-——) 2 > i Similar to Lemma [3731 we have 

1 ~ Pi 2 

the following lemma. 

Lemma 3.4. Consider N = 4, a = f} = 2. Assume (H[), 0 < P 2 < Pi < 
1, Pi + P 2 ± 0, (p5^r) 2 > § and 


0 


lin | 


1 — (1 — Ei)n 1— (1 — ejjiv- 2— \/2\ 




2aa(l —£i) 2 * 22/3(1 — El) 2 

l-(l-ei)* 2 — y/2 


/3-2 J 


7 


} with £i satisfying \3. ,5.91) 


. r 2-(l- Ml )§-(l- M2 )f 2 — v / 2 1 
mm l-5->- - A ->■ 


(3.65) 


Let {(u n ,!J n )} C M be a Palais-Smale sequence for $1^ at level ceK. Then, 
there exists a constant C, such that ||(un, u n )||]D> < C for all n € N and that 
<E> ( u n ,v n ) —> 0 in the dual space D*. Moreover, if c satisfies 

-j-S%(p 2 ) < c< ^-S%{p 2 )+ iid ^{u,v)<^-(S^(p 1 ) + S^(p 2 )); (3.66) 

iV iV {u,v)eN iv 

c ^ —S'— (mi) an d c ^ f or a ^ ^ ^ N\{0}, (3.67) 

then, up to a subsequence, (u n ,v n ) —> ( uq,vq) in D. 


Proof. Follow the processes of Lemma 13.21 and Lemma 13.31 carefully. First, by 
the similar argument as that in Lemma T3.31 we obtain (13.631) and (13.641) . Then 
the remaining work is similar to the proof of steps 3 ~ 5 in Lemma 1X^1 We can 
see that the only difference is that from (13.411) to the end of Step 3. Thus, we 
can start as the following. Since ./V = 4, a = fi = 2, we have S{p) = (1 — p)*S, 
where S is the best constant for the Sobolev inequality in K 4 . Then (13.401) and 
(13.411) are equivalent to 


(l-p 2 )~ i S< 


L 4 (R 4 


+ 20(1 -p 2 )*S 


(3.68) 


and 


(l — pi ) 4 5 < ||uo|||4( R 4) + 20(l — P 2 ) 4 S. 


(3.69) 


Consider f(t) = t 2 + 20(1 — p 2 ) i S, which is increasing in (0,+ 00 ). Assume 
0 < 2 ~/ 2 ,P 2 < Pi, we have f{\S 1 x(p 2 )) < (1 — pi)*S. Hence, by (13.691) . we 
have 

II' u °IIl 2 *(r n ) — — M 2 ) 2 S 2 = -S 2 (p 2 ). 
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Similarly, by (13.681) we have ||wo||| 2 *( R w) > (p 2 ). Thus 


| 2 * 


I wo 112s* (rN) + l|wo|li2*( R N) > -S 2 (p 2 ) + -S 2 (p 2 ) — S 2 (p 2 ), 


a contradiction to (13.381) . 


□ 


4 Nonexistence of the Nontrivial Least Energy 
Solution 


We introduce the following notation: 

l/%) ,\ „ *-(N-2)( a + ,3-2) 


C" _1 Up: 

°a,/3 — ^15(27) 


{pi) if p 2 < Hi, /3 > 2; 


: = min 




4—(jV —2 )(q!+/3 —2) 


(iO if p 2 = pi = P 


and a > 2, fi > 2. 


In particular, if V = 3,4, we define the following simpler constants 
C" )/3 := min{0.09,0.09S'-5(/ri)} if N = 3, p 2 < Pi, a > 2 ,fi > 2. 

^ :=^ifiV = 4,a = /? = 2,/z 2 < m . 

Theorem 4.1. Assume that ft > 2,p 2 < fix or a > 2, fi > 2, /z 2 = pi = p. 
Suppose that 


| either h(x ) satisfies (H i) z/a + /3 < 2*; 

| or /i(x) satisfies (H[)and p\ + p 2 0 z/a: + /3 = 2*. 

If further, B < ©o ■= C’ a ^ for respective cases of p i,/z 2 and a,/3, then the least 
energy c of the system 

c: = , in f 3-(w,w) = 

(u,v)(zAf iV 

Moreover, c is achieved by and only by (±z£b, 0), cr > 0 if p 2 < pi and by 
(±z%, 0) and (0, ± 2 #) z/ p 2 = Hi = M ^ 0 (resp. (±z CT , Xi ,0) and {0,±z aiXi ) if 
p 2 = Mi = 0). T/zai is, problem li.il) has no nontrivial least energy solution. 

Proof. First, we consider the case of p 2 < Pi- Since (0, zff 3 ) £ Af and $(0, z% 2 ) = 

i iV 

■^S~(p 2 ), we have that 

c ; = inf $( u > u ) < 

(u,v)(zAf iV 
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On the other hand, note that (zg 1 , 0) e Af and $(^ 1 ,0) = jfS* (m 1 ), we get 
that 

c ~ , <f>(u, v ) < (4.1) 

(u,v)(zj\ f iv 

Since M2 < Mi, it follows that S(mi) < 5 (/x2) and that c < If c < 

(mi), then by Lemma [2TTI we have c > 0. By Lemma IdTTl c can be obtained 
by some (0, 0) ^ (</>, x) £ AC Notice that the functional $ is even, we have 
(101,1x1) £ N and ^(0, x) = $(|<A|, Ixl), hence (|0|, |xl) is also a ground state 
for $. Without loss of generality, we can assume <j> > 0, x > 0. Moreover, <j> ^ 0 
and x ^ 0. If not, <j> = 0 implies that x ^ 0 is a solution of 

f-Au - M2jyp = \v\ r ~ 2 v inR^, 

jo^veD 1 - 2 ^), 


then $(</>, x) = 3>(0, x) > {n 2 ) > > c, a contradiction. If 

X = 0, </> ^ 0, then similarly we can get a contradiction. Recalling that 

c = W.e = (t-^)IIW.x)IIJ+(^-£)(lWb. ( «» ) + Wi'.-m). 


/ (|V0| 2 


/ |0| 2 dx + a 

[ h(x)(/) a x 0 dx , 

/RJV 

FI 

J E" 

/e» 

[ (|Vxl 2 

/RW 

x 2 \ , 

-^) <fa = 

[ \x\ 2 'dx + P 
J R» 

[ h(x)(j) o ‘x 0 dx. 
1 rjv 


Next, we discuss two cases according to the sign of f RN h(x)<j) a x 0 dx. 

• Case 1: [ h(x)cj) a x 0 dx < 0. 

It follows that 

/ (|Vxl 2 — ^ 27 ^ 12 )^ < [ \x\ 2 *dx, 

J E» F| Vr" 

which implies that llxll 2 2 *( R n) > <S'^"(M 2 )- By the Hardy’s inequality and the 
Sobolev’s inequality, we have that 


*<**> = <5 "ST? 


) 11(0, X)llro + ( 


a + /3 


2*) (ll0llx,2* 


(i») 


llxll 2 


L2*(RJV) 


M2 


+ “ 2^) llxllia-(RJW) 


)S(m 2 )||xII^ ( rW) + - ^JllxIliV, 


1 / , 1 , , 

> 2 (1*2) > (Ml) > c, 


a contradiction. 
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• Case 2: / h(x)(p a x^dx > 0. 

Jr n 

Notice that 

C = &(</>, X) = ^(ll0llia* (R w ) + ||xlli’a* ( RW)) 

+(« + /3)(i-T«) / h (x)<P a X 0 dx 

2 a + P Jrn 

< ^(/h), (4.2) 

we have 

llxllL 2 *(R iv ) < & 2 (Ml); II<^lIZ,2* (rJV) < S 2 (/il)- (4-3) 

By the Hardy’s inequality (or the Sobolev’s inequality) and the Holder’s in¬ 
equality, we see that 


“S'0^2) ||x lz,2*( R JV) 

< 

llxl& 



= 

11x111*2. (RW) +/3 f h{x)<j> a x p dx 



< 

llxlli*2*( RJV) +/3 [ h + (x)(j) a x^dx 

y 1 Jrn 



< 

||xllf 2 * ( RN ) +/?05^“(Mi)||xlll2. (RW) . 

(4.4) 

Denote cr = Hxll^^jv), then 



S(J* 2 ) 

< 

(T 1 -^ + /30S^“(/ii)<7^. 

(4.5) 


2 \t _o 8 _2 

Let /(cr) = ct 1- ^ + P@S~— . Since /3 > 2, we have that /(cr) 

is increasing in (0,+oo). If /x 2 < /ti,/3 > 2,0 < C' a p, the direct calculation 

implies that /(Stt (/ii)) < S(/j, 2 )- Then (14.51) implies that llxll^mjv) > 5* (/ii), 

\ n 

a contradiction to (14.31) . By the above arguments, we obtain that c = — S’— (/ti) 
and obviously it is achieved by 0), a > 0 . 

Let ( 0 , \) be a minimizer of c = jjS~ (/ti). If 0 ^ 0 and % ^ 0, it will 
lead to a contradiction if repeating the above arguments. Here, we may assume 
0=0, then $( 0 , x) > 2 ) > ^■S' _2 "(/i 1 ) = c if /t 2 < /ti- Hence, c is only 

achieved by (0,0), where 0 is a weak solution of 

f-Au - /iipp = |m| 2 * _2 m inR^, 

[O^tieD 1 ’ 2 ^). 

By [31], it is easy to see that c is only achieved by (±z£, 0), cr > 0. 
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Second, we consider the case of /r 2 = Hi = p, a > 2, /3 > 2, 0 < C' g. The 
difference is the computation of f[^S^(p i)): 

= i) + ^e(^5 (w_2,( ^' 2) ( Ml ) 

< S(/r 2 ) = S(/ii). 


1 


Then (14.51) implies that ||xIIl 2 *( R jv) > -S' 2 (p). Similarly, we can obtain that 




Then 

O* ry* N N 

llxll L2*( R N) + ||</>|| £2*( R iV) > S 2 (/i) = S 2 (Ail), 

a contradiction to ED. Finally, it is easy to show that c is achieved by and 
only by semi-trivial solutions (±z^,0) and (0,±zg) if p ^ 0 (resp.(±z CT;Xi ,0) 
and (0, ±z aiXi ) if p = 0 ). □ 

Proof of Theorem 11.11 It is a straightforward consequence of Theorem 14.11 

□ 


5 The Existence of Mountain Pass Solutions 


If min{a, j3} = 2, we denote 


C, 


a,/3,/ii,/x 2 


lin | 


S(Mi) 


S(/i 2 ) 


2S^(p, 2 )’ 2S^(p 1 ) 


} 


(5.1) 


Remark 5.1. We observe that C a jj All , /t2 > min{0.3,0.3S2 (/ki)} if N = 3; 
C a ,p, mttla > ^ ifN = 4 and a = /3 = 2. 

Remark 5.2. For a > 2, we can see that the indefinite sign of h{x) has no 
effect on the proof of Theorem 2.2 in W- Thus, for all a > 0, zjf 2 is a local 
minimum point of $ in A f. Denote 

K , = {u 6 D ^( R N ) \{ 0 } : ||«||= 2 = 

which is the corresponding Nehari manifold of If 2 , where If is defined in t3.2\) . 
For a = 2,^2 > 0, we can obtain that (for the details we refer to f 1, Theorem 
2.2] or JJ5[ Lemma 3-4]) (4>,z£ 2 + x) € Af, where z£ is defined in 11.51) . Let 
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t > 0 satisfy t(zf 2 + x) G Af^ 2 , then 


$0, *a 2 + X) ~ m t(z% 2 + x)) 

= [ A(x)|^| 2 |^ 2 +x| /? )( 1 + °(1)) 

> (^(m)||</>|| 2 2 . (RW) - €>110111,2* (JRiV) ||-^ 2 +Xlli2* (R JV } )(l +0(1)) 

= (^ 5 0l) “ 0 II~£ 2 +Xlli2* (R iV ) )ll^lli2* (R iV ) (l +0(1)) 

^ — 4 II^IIl2*( R jv)( 1 + o(l)) 

as II(05 X)IId> y 0- Thus, when 0 < C a , )l g )|Uli(U2 (see i l5.il ) ), we have that 

±s(p 1 )-es^(p 2 )> o. 


It follows that 

$«>, + x) - *(0, t{z? + x)) > 0 (5.2) 

provided that (4>,z£ 2 + x) G Af is sufficiently closed to (0, z£ 2 ). On the other 
hand, since z£ 2 is a minimizer of I^ 2 on A /^ 2 , we have 

$(0, t(z£ 2 + x) - m z^)) = J w (<(^ 2 + x)) - WO > 0. (5.3) 

From \5.2\) and \5.S\) . we conclude that $(</>, zg 2 + x) ~ ^(O,^ 2 ) > 0 for any 
(<fi, zf: 2 +x) G Af sufficiently close to (0, zf 2 )(with strictly inequality hold outside 
the manifold {0} x Z^ 2 , where Z^ 2 = {z£ 2 ,cr > 0} ), i.e., (0 , zf 2 ) is a local 
minimum point of $ inAf. Similarly, fora = 2,p 2 = 0, we replace z£ 2 by z a ^ Xi , 
where z a}Xi = cr'^ ),a > 0 ,x t G R N , and z a (x) = z afi , Zi,o{x) = 

[N(N - 2)]0f^ 

- N _ 2 , we can obtain that &((f>,z aXi + x) — 3>(0 ,z aXi ) > 0 for any 

[l + |x| 2 ] — 

(<j>, z a ,xi+x) G Af sufficiently closed to (0, Z c , jXi ) (with strictly inequality holding 
outside the manifold {0} x Zq, where Zq = {z a ^ Xi ,a > 0 ,Xi G R N }), i.e., 
(0, Zcr,xi) is a local minimum point of $ in Af ■ Similarly, if (3 > 2 or (3 = 2 with 
0 < Ca,p,in,u 2 ( see ( EH ) ), we can obtain that (zff,®) is a local minimum. So 
does (z atXi , 0) if pi = 0 and either (3 > 2 or (3 = 2 with 0 < C ai( g i|UliAl2 . □ 


By Remark [5721 the semi-trivial solutions (0 ,z% 2 ) or (0 ,z a , Xi ) turns out to 
be local minimum points for the functional <f>| .^, which consequently exhibits a 
mountain pass geometry. 

Next, our goal is to construct a mountain pass level for the functional on the 
Nehari manifold at which the Palais-Smale condition holds in view of Lemmas 
13.21 13.31 and 13.41 For the simplicity, when p = 0, we also use the notation 
Zi instead of z\fi. When a > 2 or a = 2 with 0 < C a> p tflli ^ a (see (15.11 ) ), by 
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Remark 1 5.2 1 we know that (z^ 1 ,0) is a local minimum. Similarly, when (3 > 2 or 
(3 = 2 with 0 < (0,zf 2 ) is a local minimum. By Theorem 14. II when 

M 2 < Mil© < C' b or /j 2 = Mi,0 < we have inf 4>(u,u) = -^“(mi)- 

{u,v)eN 

Since $ is even, for (u,v) G J\f, we have (|u|,|u|) € A/ - . Thus, (|u|,|u|) G 

A/'flA/' and $(|u|,|u|) = $(|u|, |u|) = ${u,v), then inf <I>(m, v) < — (mi). 

(u,v)eJT N 


Assume 


inf $(u,u) < 
i,v)(3f7 


In 1 n _____ 

— S~ (fJ, i) < — S'” (^ 2 ), similar to Lemma RTTl we can 


prove that the infimunr can be achieved by some (uo,vo), and the nrinimizer is 
a critical point of $. It is easy to see uq > 0, vq > 0, thus (uq, vo) G AT C\J\T and 
$(uo,vo) = $(uo,r , o) > jjS~r (fii), a contradiction. Thus, we have 


inf ®(u,v) = 

(u,v)eJ7 A 


(5.4) 


when M 2 < Mi 1 © < Cq,/ 3 - 

Next, we consider the set of the paths in AT joining (zg 1 , 0) with (0, z(( 2 ), 
i.e., _ 

7 = ( 71 ; 72 ) : [ 0 , 1 ] —> AT continous such that 

7 i(0) = 0,7i(l) = zg 1 , 72 ( 0 ) = z% 2 , 72(1) = 0 
and define the associated mountain pass level 

Cmp '■= inf max $( 7 ( 1 )). (5.5) 

7eStG[0,l] 




5.1 The case of TV - 3, \ < 5t(^ 2 ) + S%(tn) < St. 

If M 2 < Mii we define 


Mi 1 — ^(1-4 W ) 

[(l_4M 2 )i+(l-4Mi)i]i 

(5.6) 

:= 2(1 - 4 W ) _ 

1 - 4m2 

(5.7) 

M 3 := - min {M 1 ,M 2 }, 

(5.8) 

then we have 


2 n „, .,S(n 2 ) + 2 n 1 + M 3 n 

M z) ( 2 ) 2 >/ 2 W> n ^ 2 (M 2 ). 

(5.9) 

If M 2 = Mi = Mi we define 

M 3 := 

(5.10) 
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then we have 


|d _ m 3) (^i l±2Mf > (w) . 


Define 


M 4 := 


1-3(1- M 3 )^ 

1 (JV- 2 ) (a + 3 - 2 ) 

( a + /3) ( 2 ) 


> 


V2 

12 ' 


(5.11) 


(5.12) 


Lemma 5.1. Assume N = 3, a > 2,(3 > 2, a + (3 < 2*, 1 < such 

N N N 

that S'* ( 1 Ki) + S'* (n 2 ) < S’" 5 ". The weight function h(x ) satisfies ( H 2 ) ami 
f (^0 if a+ 13 <2* 

\(^0 If a+ (3 = 2* 

,fi 2 ( see fh.H ) ), where 0 is defined in HI.12)) . Then, if® < min{M 4 , C' a p}, 
the functional $ exhibits a mountain pass geometry and the mountain pass level 
satisfies Vi. 1 ()\) and 17.111) . 


. Moreover, if min{a,/3} = 2, we assume that 0 < 


Proof. We claim that when a < M 4 , we have 


>(7ft)) > -^(1- M 3 )( ,g ^ 1 - > + ,S '^ 2 ^ ) 2 for all 7 GS. (5.13) 
Let (71,72) G £, since (71 (t), 72 (t)) G A 7, we have 


max <1>( 

te[o,i] 


/ K. N 


|V7i(t)| 2 d.T-/ii / -T~w-dx+ f \V^ 2 (t)\ 2 dx - H2 
Js. N \ x \ J R N 


r 7 m 
U m 2 


dx 


Ir n 


(7i {t))+dx+ (-/ 2 {t)) 2 dx + (a + (3) h(x)(^{t^dx 


/R w 


and 


< h(7iW,72(i)) = / (71 (t)) 2 dx+ j ( l2 {t)) 2 dx) 

xi y j RJV + J R N + / 

a + (3 — 2 


H^i'yii.ty+hziW+dx (5.14) 


or 


or 


^(7iW,72(*)) = 11(71 (*)’72(0)11 


k a + (3 


2*1 


( / (7 iW)+ete+ f (7 2 {t)) 2 dx) (5.15) 

V JRJV ^ Jrw ^ 1 


$(7iW.72(*)) = II (71 (*) 5 72 (0) lli> — (1- — 5 J-) 

[ h(x) (71 (1)) (72 (t))^_dx. 


(5.16) 
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Denote /*(t) = f RN (7,:(£)) + da; for i = 1 , 2 , then 

/i(0) = 0 < f 2 ( 0) = [ (z£ 2 ) r dx and /i(l) = [ (z ^ 1 ) 2 *dx > / 2 (1) = 0, 
Jr n Jr n 

hence, by the continuity, there exists some to € (0,1) such that /i(to) = f2(to) > 
0 . From the definition of S(fj, 1) and S(/j, 2 ) and the Holder’s inequality, we have 


S(Vi)( [ (71(^0))+) 2 +S(n 2 )( [ (72(^0)) 

'ii» J X J t» 


2 *\ 2 * 
+ 


< 


/ R N 


|V7i(t 0 )| 2 da; - m j 

Jr 


7i(*o) 


dx 


R N \ x \ 


\^j2(t 0 )\ 2 dx - 


M 2 


dx 


Jr n 


f 7lfa) 

/RJV kl 2 


(71 (*o ))\dx+ j (j 2 {t 0 )) 2 + dx + 


Jr n 


(a+ / 3 ) / /i(x) (71 (t 0 ))“(72(^0))+^ 


/R^ 


< 


Jr n 


('n(to)) 2 ,dx+ / (72(^0))+d»H- 


Jr n 


(a + /3)e( f (71 (t 0 )) + dx ) 2 ( f (^(to )) 2 dx) 2 

K JR N ' E" ' 

Set a = J rN (7i(fo)) + dx = J rN (7 2 (io)) + dx, then we obtain that 

(S(ni) + S(li 2 ))(7^ < g + (a + /3)0 ct^^ 

and it is equivalent to 

S(n 1 ) + S(^ 2 ) < + (a + /3)0er^ . 


( 5 . 17 ) 


( 5 . 18 ) 


Denote g(a) a 1 & + (a + / 3 ) 0 ct — 2* , which is increasing in ( 0 , 00). Recall 

that 0 < M4,, it follows that 


s(( 1 - m 3)( 


S(fi 1) + S'(M2) \ § 


N \ 

> T ) 


= (l-M 3 )» S fa) + S(w) +(a + 


'S'(Mi) + S(M2) 


(c» + /3-2)(JV-2) 


< S(ni) + S(fi 2 ), 
which implies that 


< 7 > (l- M » )( gW+£W )f, 
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then 


$(7 (to)) > 
> 


> 




. 2 a 


a + /3 


2 [(?--h?) (1 - Mj) * ( S(f ‘ l) 2 S(w) ) f 

L 2 a + p A 

+i ^h - ? )(1 “ ^w sm+ 2 sm )* 

lv-m SM+ 2 SM )*. 



we prove the claim (15.1311 and obtain that 


Cmp > ^^( M2 ) = (1 + M 3 )i>(0, zg 2 ), 

and hence $ exhibits a mountain pass geometry. In particular, Cmp > (mi) 

if M 2 < Mi; C M p > if M 2 = Mi- 

Next, we consider a special path 7 (f) = /c(f)( 71 (f), 72 (f)) £ S,1 € [0,1] with 
71 (f) = t^zg 1 and 72 (f) = (1 — f) 5 z £ 2 , where fc(f) is a positive function such 
that A:(f) ( 71 (f), 72 (f)) G A/'flA/’. By the definition of the Nehari manifold, fc(f) 
is well defined and unique. For the simplicity, we set 


a --=\m\l= [ \zZ 1 \ 2 *dx = S%(^) 

and 

b:=\\zS% 2 = [ \z^fdx = S^(n 2 ). 
Js. N 

Since &(f)( 7 i(f), 72 (f)) G A/’flA/", then by (# 2 ), we obtain 

ll(t5^,(i-f)^^)HS 

= fc 2 *~ 2 (f)((l - f)T"a + t^~b) 

+(a + /3)fc a+/3 - 2 (t)(l-t)tff / 

> fc 2 * _ 2 (f)((l - f)^ - a + t^b) for 0 < f < 1. 


Hence, 


k(t) < 


(1 — t)a + tb ' 

- (1 -f)Ta + fV b- 


TV — 2 
4 


for all 0 < f < 1 


and fc(0) = k(l) = 1. Combine (# 2 ) and (15.161) . it follows that 


( 5 . 19 ) 


®(k(t) ( 71 (f), 72 (f))) < 


fc 2 (f) 

TV 


((1 — t)a + tb) for all 0 < f < 1 , 
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hence. 


Cmp = inf max $ (7(f)) <max<^(k(t)(j 1 (t),j 2 (t))). (5.20) 


By (|5.16p and (15.191) . we have 


&(k(t)(7i(t),72(t))) < 


1 r (1 — t)a + tb 


N 


(1 — t) 2 a + t 2 6-1 


((l—t)a+tb) for all t € ( 0 , 1 ). 


After a direct computation, the right-hand side achieves its maximum at t = \ 
and the maximum is -^(a + b). Combining with (15.201) . we obtain that 

_ 1 , TV N \ 

C M p < max $(fc(f)( 71 (f), 72 (t))) < —(S^(m) + S^(fj, 2 ))- (5.21) 

te[o,i] A 

Recall that | we have (M 2 ) < 2(m 1 ). Therefore, from the above 

arguments we obtain that 

1 ~\~ N 2 N 1 TV N 

2 ) < -S^(n 1) < Cmp < ^(S T (m) + 2)) 


. r 1 n K 3 jv 

<7 min{-S ^ , —S 2 (/ii)} if H2 < Hi] 

1 -\- Ad'l N 2 N 1 AT 

— -jj —(m) < C M p < -jyS* (m) < —S^ if M 2 = Mi = M- 
Combining (15.41) . it follows that both (13.101) and (13.lip are satisfied. 


and 


□ 


5.2 The case of iV = 4, \ < (j ^) 2 , S * (/I 2 ) + 5^ (mi) < S*. 

In this case, we only consider a = (3 = 2. If M 2 < Mi 1 define 


M[ := 1 - 

4(1 — Mi ) 5 

(5.22) 

[(1-M2)*+(1-Mi)*] 2 ’ 

M' 

1 — M 2 

(5.23) 

m' 3 

:= ^ min{M(, M^}, 

(5.24) 

then we have 



2 (1 M^ SM + S(n] 
N y 3A 2 

lA 2 n 1 + M 3 n_ 

-) > ]v ^ 2 (Mi) > ^ 5= (m 2 ). 

(5.25) 

If M 2 = Mi = M: we define 

Mo := 

3 4’ 

(5.26) 
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then we have 


N 




S(p 1 ) + S(p 2 ] 


y 


1 + M'l n N_ f v 

> -^^ S2 (^)- 


(5.27) 


Define 


M' := 


l-I(l-M')^ 

i (»- 2 )W- 2 ) 

(a + /3)(i) 


(5.28) 


If a = /? = 2, then M 4 = I — 1(1 — Mg) 2 > I. Similar to Lemma 15. 11 we have 
the following 


Lemma 5.2. Assume N = 4, a = = 2,1 ) 2 such that S 2 (p 1 ) + 

AT IV 

S^ {p 2 ) < S'" 2 ". Suppose that the weight function h(x) satisfies (H 2 )-(H' 1 ) and 
that 0 < Cq,,/ 3 lAtllA i 2 (see 15.11 ) ). If moreover, 0 < min{M 4 , C' a p}, then $ 
/ias o mountain pass geometry and the mountain pass level satisfies \3.66\) and 

m . 


Proof. Since we always assume a > 2, fj > 2, a + ft < 2*, when N = 4, the only 
possibility is that a = fi = 2,a + /3 = 4 = 2*. Thus we required (H[). Note that 

\ 2 implies (p 2 ) < 2 (pi). We can follow carefully the processes 

of Lemma T5. II and obtain the results. Here we omit the details. □ 


5.3 The case of IV = 3, § < ±5|g,2( 

In this case, if p 2 < Pi, M 3 is redefined as 

If. V2 


S(n 2 )+S(m) \ -2 


iv 

) 2 > S 


M3 := 




1 - 


[(1 - 4 ^ i )3 + (1 - 4 / i 2 ) 3 ] 2 


where Mi, M 2 are defined in (15.61) and & Then (15.91) satisfies and 

2(l-M a )(3M+j3»i)f >sf 


If p 2 = pi = p , M 3 is redefined as 


1 


M 3 := - - 


1 


2 4(1-4 pY 

then (15.111) and (15.301) are satisfied. Here we define 


M 4 := 


1-|(1 - m 3 )^ 

1 

( a + P){ 2 ) 


> 


V2 

12 ' 


(5.29) 


(5.30) 


(5.31) 


(5.32) 


Then we have the following result: 
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Lemma 5.3. Assume N = 3 ,a > 2,(3 > 2, a + [3 < 2*,i < such 

that 2( S^ 2 ^ + J-) ) 2 g— _ y/jg weight function h(x) satisfies ( H 2 ) and 

{ ( 1 ) / +/? < .In particular, when min|a,/3i = 2, we assume that 

(H[) if a+ (3 = 2* X S 

® ^ ,fi2 (sCC ED ) ). Then, if 0 < C' a ^}, $ ftas a mountain 

pass geometry and the mountain pass level satisfies both 113.10 1) and \3.11\) . 

Proof. Analogous to Lemma 15.II $ exhibits a mountain pass geometry at level 
Cmp satisfying 

2 (1 _ M3) ( ^ l) + ^2 ) )f < CMp < j _ s %( Ml) + 


N 


N 


where M 3 is defined in (15.291) for p, 2 < Mi or (15.311) for fi 2 = Mi = M- Moreover, 
by (15.301) . we have 

^ < Cmp < ^^(/ir) + 

It follows that (^ 2 ) < (pi). If p 2 < Mi> then we have 

1 Ad a N 2 IV 3 N 

^-^(/x 2 ) < 1) < Cmp < jjSTfa); 

if ^2 = Mi = Mj we get that 

1 1 + M 3 N 2 JV 

— S 2 (mi) < —-— S 2 (M 2 ) < C M p < j^S 2 (mi). 


Thus, Cmp satisfies (13.101) and (13.111) . 


□ 


5.4 The case of N = 4, \ < (ffg) 2 ,2( g(/J2) + S(An) ) 2 > £t 
In this case, if p 2 < mi> M 3 is redefined as 

1 


M' :=-{m(,M',1 


[(1-Mi)« + (1-M2)«] 2J 
where M[, M( are defined in (15.221) and (15.231) . Then (15.251) is satisfied and 


(5.33) 


2(1 - Mi)( A bU+M )* > st. 


If M 2 = Mi = M> M 3 is redefined as 


M < - 1 1 

M3 '“2 4(1 — 4m) ’ 


(5.34) 


(5.35) 
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then (15.271) and (15.341) are satisfied. Now M 4 is defined by 


M' := 


1 (iV-2)(c. + g-2) • 

(a + /3)(i) 


(5.36) 


Similarly, we have the following result: 

Lemma 5.4. Assume N = A,a = B = 2, — < (-—) 2 such that 

2 1 - M2 


2 /5V2)_+SVi)\f > s n 


Suppose that the weight function h{x) satisfies ( H 2 ) and (H[); 0 < 

(see mv). Then, if 0 < min {M^,C' a g}, $ has a mountain pass geometry at 
the level c satisfying both \3. 66\) and \3.67\) . 

Proof. It is analogous to the proof of Lemmas 15.21 and 15.31 □ 

Based on the results of Lemma l3.2l ~ Lemma l5Nl we can obtain the existence 
of mountain pass solution to problem CD- 


Proof of Theorem 11.21 Define 


di :=min { 0 . 355 (^ 0 , 10- 3 [(1-4m!) 5 - (^) § (l - 4^) f ], 5 x 10“ 4 } 


y/2 9 9 


. r 3 3 1 V2 „ - A 

= mm<—, —d 2 rt 1 ,—,-,- S 2 u 1 , 

110 10 1 12 ’ 100 100 ^ h 

10- 3 [(l-4 Ml )l- (|) S (l-4/X2) f ],5x 10- 4 }. 

Then d\ < M 4 , C' a g, C\, C 2 }. In particular, the assumption 

0 < A. 


• Case I: S% (/x 2 ) + (m) < S%. 


If min{o:,/3 = 2}, then 0 < d\ implies that 0 < C a j g )#11) ^ 2 . Furthermore, 
0 < d\ => 0 < min{M 4 , C' a 0 }, 


where M 4 is defined in (15.121) . Then by Lemma l5Jl $ exhibits a mountain pass 
geometry and the mountain pass level satisfies (13.101) and (13.111) . 

. Cas e II:2( S( " l) + Sfe) ) f > S*. 

For this case, M 4 is defined in (15.321) . Analogously, by Lemma IOI $ has a 
mountain pass geometry with energy level satisfying both (13.101) and (13.111) . 
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For either case I or case II, by the Mountain Pass Theorem, there exists a 
sequence {(u„,t) n )}neN C Af such that 

$(u n ,u„) -*• C M p , ($)'|^ r (u„,u n ) -)• 0 and 

-rr, , 1 + M3 JV 

<F(u n , u„) > ———S 2 (/z 2 ). 

Recall that 0 < d\ implies that 0 < min{Ci, C 2 }, where 61,62 are defined 
in m and m - By Lemma [3.21 {(u n , u n )} rlg N admits a subsequence which 
converges strongly to a critical point (uo,vo) of $|-^, which is also a critical 
point of >I> in D. We observe that Uo > 0 ,Vq > 0, uo^o ^ 0, hence (uo,vq) is a 
critical point of $ in D. That is, ( ug , vo) is a nonnegative mountain pass solution 
of problem m- ° 


Proof of Theorem 11.31 Define 

Mi + M 2 - (ah + ah) 2 


:= min 


,10- 3 [(l-4 Ml )i-(i) § (l-4M2) § ], 


in {" 

5x10- 4 ,1^(mi)} 

= min | 

r in _4 3 3 i a/2 9 9 a. 

5x10 ,—,— S 2 (iti),—,-,- S 2 (mi) I. 

’ 10 10 K ’ 12 ’ 100 100 


10 -a [(1 _ 4(ti) | _ ( 1) I (1 _ 4w) I], 


Then 


c ?2 < min 


“fa 


l-(l-ei)w l-(l-er) 


22 a(l - ei)- 2 *- 22/3(1 —£ 4)2 


—, Ci, C2, C at p tlll tfl2 , M4, < 7 a d, 

71“ J 


and (U.16l) =» 0 < d 2 . Similar to the proof of Theorem ll.21 based on the results of 
Lemma l5.1l and Lemma l5.3l when 0 < d 2 , we obtain that 4> has a mountain pass 
geometry which energy level satisfies both (13.101) and (13.111) . By the Mountain 
Pass Theorem, there exists a sequence {(tin, u„)} ne N C Af such that 


—> 0 and 


, 1 + M 3 N 

$(u n , V n ) > -—- S 2 (M2)• 


Notice that 


0 < d 2 => 0 < min 


in { — 


l-(l-£l)« 1 (1 £fi) ~ 


2 2 ct(l — £ 1 ) 2 * 2 2 /3(1 — £\)^ 


-,C 1; C 2 }, 


where £1 satisfies (13.521) and Ci, C 2 are defined in (ESI) and m • By Lemma 
13.31 {(« n Mn)}n6N admits a subsequence which converges strongly to a critical 
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point (u o,vo) of which is also a critical point of $ in HJ. Also we know 
that uq >0,vq> 0,uqVq jk 0, hence (uo,Vo) is a critical point of $ in ID. That 
is, (uo,vq) is a nonnegative mountain pass solution of the problem ( 11 . 11 ) . □ 


Proof of Theorem 11.41 Define 

2 - (1 - Ml )§ (1 - M2 ) § 2 — \/2) 


g ?3 := min j 
= min | 


2 — (1 — Ml )i — (1 -fjL 2 )% 2 — \/2 1 2 — \J2 

8 ’ 4 ’ 4’ 2 ’ 4 /' 


Then d 3 < min | 


2 -(l- m )l-(l-/x 2 )i 2 — \J2 


•^4) C a ,/3’ ^ fmd 


(I1.19D => © < c? 3 . Thus by using Lemma [3~4l Lemma IOI and Lemma IOI the 
problem (11.171) has a nontrivial weak solution {uo,vo) such that uq > 0,i>o > 
0, uqVo ^ 0. □ 
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